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Abstract 

Distributed  Mission  Training  (DMT)  enables  participants  to  perform  within  a  virtual  battlespace 
created  through  networking  of  several  high-fidelity  simulations.  In  a  recent  exercise, 
ROADRUNNER  ‘98,  several  agencies  interacted  to  create  several  battlespace  missions  in  which 
“friendly”  fighter  aircraft  and  command  and  control  crewmembers  participated  as  trainees,  while 
supporting  roles  and  enemy  forces  were  either  played  by  operational  personnel  (virtual  players) 
or  created  by  intelligent  agent  technology  (constructed  forces).  Thus,  trainees  participated  in 
complex  demanding  war  scenarios  without  the  usual  constraints  of  cost,  safety,  and  security 
normally  associated  with  live-fire  training.  ROADRUNNER  ‘98  was  sponsored  by  the  Air  Force 
Modeling  and  Simulation  Office  (USAF/XOC)  with  the  Air  Force  Research  Laboratory,  Human 
Effectiveness  Directorate’s  Warfighter  Training  Research  Division  (AFRL/HEA)  serving  as 
program  managers  and  the  Theater  Command  and  Control  Simulation  Facility  (TACCSF)  as 
systems  integrators.  Extensive  support  was  received  from  the  Training  Office  at  Air  Combat 
Command’s  Deputy  for  Operations  (ACC/DOT),  the  Air  Force  Information  Warfare  Center’s 
Advanced  Combat  Simulations  Division  (AFIWC/SAM),  the  Airborne  Warning  and  Control 
Systems  (AWACS)  training  office  at  the  522nd  Air  Control  Wing,  the  107th  Air  Control  Squadron 
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(ACS)  of  the  Iowa  Air  National  Guard  (IANG),  and  the  Naval  Air  Warfare  Center’s  Training 
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operational  pilots,  Weapons  Directors  (WDs),  and  Air  Surveillance  Technicians  (ASTs)  from  the 
522nd  ACW,  Tinker  AFB,  OK,  27th  Fighter  Wing,  Cannon  AFB,  NM,  The  185th  and  132nd 
Fighter  Wings  of  the  IANG,  33ld  Fighter  Wing,  Eglin  AFB,  FL,  and  from  the  United  States  Air 
Force  in  Europe  (USAFE). 

This  paper  describes  the  process  by  which  C3  teamwork  dimensions  were  identified,  measures 
formulated,  and  the  rating  instrument  refined.  Measures  of  AW  ACS  team  performance  were 
developed  based  on  previously  collected  cognitive  task  analysis  data  and  refined  through  the  use 
of  focal  groups  comprised  of  operational  AWACS  WDs  or  WD  Instructors. 
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1.  Distributed  Mission  Training:  Background 

Distributed  simulations  have  become  an  increasingly  essential  aspect  of  operational  military 
training.  As  both  training  and  defense  budgets  shrink,  ways  of  reducing  the  number  of  flight 
hours  and  yet  maintaining  optimal  levels  of  expeditionary  force  readiness  are  sought. 
Distributed  Mission  Training  (DMT)  is  looked  upon  as  a  way  that  will  allow  warfighters  to  train 
in  a  manner  consistent  with  Air  Force  Doctrine  60:  Train  like  you  fight,  fight  like  you  train. 
Previous  constraints  (e.g.,  safety,  flying  status,  flying  hours,  aircraft  and  environmental  factors) 
have  hampered  training  efforts  from  following  this  philosophy  to  its  fullest  capacity.  DMT 
addresses  these  constraints  and  allows  the  warfighters  of  our  nation  to  perform  crucial  training 
missions  with  maximum  proficiency  and  safety,  at  lower  cost.  General  Fogleman,  Chief  of  Staff 
Air  Force,  stated  on  29  Jan  1996: 

I  am  convinced  modeling  and  simulation  technologies  available  today  will 
enable  us  to  significantly  change  the  way  we  train  in  the  future.  We  are  at  a 
crossroads  where  simulator  technology  today  will  be  critical  in  the  success  of 
our  effective  use  of  follow-on  weapon  systems  ....  We  need  to  take  a  hard  look 
at  how  this  technology  will  change  our  training  philosophy  as  well  as  how  we 
develop  future  weapon  systems. 

DMT  is  focused  within  a  synthetic  warfighting  environment  that  grants  a  multitude  of 
advantages  for  more  accurate  observation  and  assessment  in  areas  that  were  previously  difficult 
to  research.  Performance  can  be  modeled  within  and  across  individuals,  teams,  squadrons,  and 
complex  functions.  Linkages  among  diverse  interdependent  units  can  be  identified  and  modeled 
to  achieve  greater  efficiency.  This  intensive  modeling  of  performance  within  a  complex  system 
enables  a  more  precise  assessment  of  the  impact  of  interventions,  such  as  training  technology, 
cause  and  effect  relationships,  and  changes  in  the  distribution  and  display  of  information. 

DMT  offers  flexibility  that  stretches  worldwide  by  linking  participants  across  network  lines, 
allowing  individual  warfighter  training  to  occur  concurrently  in  simulators  whose  presentation  of 
a  commonly  flown  mission  occurs  in  real-time.  This  capability  supplements  cockpit  flight  hours 
with  a  synthetic  reproduction  of  a  complete  and  complex  warfighting  environment  that  combines 
realistic  scenarios  with  simulated  operational  systems.  The  focal  point  of  this  paper  is  the 
Airborne  Warning  and  Control  Systems  (AW ACS)  segment  of  a  Distributed  Mission  Training 
demonstration  ( ROADRUNNER  ‘98).  This  exercise  tied  AW  ACS  weapons  directors  from 
Tinker  AFB,  OK,  F-16  aviators  from  AFRL,  AZ,  F-15  aviators  out  of  Kirkland  AFB,  NM,  and 
an  A- 10  aviator  from  AFRL,  AZ  together  for  one  week  of  DMT  training. 

2.  USAF  AWACS  Teams 

All  command  and  control  teams  share  some  common  task  and  performance  characteristics. 
These  teams  perform  in  highly  interdependent  roles,  tracking  and  coordinating  some  type  of 
tactical  action,  in  a  manner  consistent  with  overall  strategic  goals  and  procedures,  for  a  defined 
sector  of  air  and/or  land  space  over  a  sustained  period  of  time. 


The  AW  ACS  team  serves  within  this  team  definition  to  augment  the  need  for  airborne 
surveillance  and  command,  control,  and  communications  functions  for  tactical  and  air  defense 
forces.  This  provides  the  means  to  detect,  identify,  track,  and  intercept  airborne  threats.  The 
aircraft’s  god’s-eye  capability  offers  an  altitude-independent  360°  view  of  more  than  200  miles 
over  both  land  and  water.  There  are  five  mission  crew  categories  related  to  the  function  of  the 
AW  ACS  crew:  the  Mission  Crew  Commander  (MCC),  Senior  Director  (SD),  Weapons  Director 
(WD),  Air  Surveillance  Officer  (ASO),  and  Air  Surveillance  Technician  (AST).  AW  ACS  teams 
typically  work  in  teams  of  two  to  four  crewmembers  as  part  of  the  airborne  command  and 
control  group  located  within  E-3  Sentry  aircraft. 

The  ROADRUNNER  ‘98  exercise  focused  primarily  on  the  integration  of  multiple  warfighter- in- 
the-loop  training  simulators,  combining  them  with  synthetic  forces  to  form  a  virtual  battlespace 
to  conduct  several  assorted  combat-oriented,  tactical  training  exercises.  The  overall  goal  of  this 
effort  was  to  compare  initial  team  performance  with  subsequent  performance  after  participation, 
and  give  the  participants  the  opportunity  to  critique  both  the  systems  and  the  concept  of  training 
(Parton,  1998).  Each  scenario  was  based  on  intelligence  briefings  and  air  tasking  order  (ATO), 
and  each  fighter  and  AW  ACS  team  planned,  coordinated,  and  briefed  their  mission.  The 
combined  teams  flew  the  mission  in  their  respective  simulators  with  a  trusted  agent  monitoring 
and  evaluating  their  performances.  The  teams  and  instructors  participated  in  an  after-action 
review  aided  by  a  mission  playback  system  that  assisted  during  the  debriefing  process.  Each 
team  participated  in  7  separate  missions  over  the  course  of  one  week. 

3.  Development  of  Team  Observer  Form 

The  AW  ACS  crew  coordinates  communications  received  from  a  number  of  sources,  such  as 
other  WDs,  the  SD,  air  surveillance  technicians  (ASTs),  electronic  combat  officers  (ECOs), 
intelligence  operations,  base  operations,  and  friendly  pilots.  To  accomplish  this,  they  must 
exchange,  interpret  and  effectively  weight  information  and  optimize  resource  allocation 
decisions  across  team  members,  over  time,  and  under  stress  and  fatigue.  These  decisions  regard 
shared  resources,  such  as  surface-to-air  missile  sites  and  various  combat,  reconnaissance, 
refueling,  and  search  and  rescue  aircraft.  Relevant  information  must  be  distributed  to 
appropriate  personnel  and  updated  over  time,  in  dynamic  conditions  which  may  require  shift 
changes  in  personnel.  Information  is  often  verbal,  and  may  be  missing,  degraded,  passed  along 
from  unfamiliar  sources,  or  misinterpreted  by  others.  In  addition,  information  is  often 
communicated/interpreted  by  individuals  with  only  partially  overlapping  awareness  of  the 
battlespace.  For  the  ROADRUNNER  ’98  study,  assessment  of  AW  ACS  team  of  performance 
was  based  on  SME  ratings  of  several  dimensions  of  AW  ACS  team  performance.  These 
dimensions  were  generated  based  on  cognitive  task  analyses,  followed  with  focus  groups 
comprised  of  experienced  AW  ACS  instructors.  These  focus  groups  served  to  refine  the  rating 
form,  and  included  the  SMEs  who  actually  participated  as  ROADRUNNER  ’98  trusted 
agents/evaluators. 

Cognitive  Task  Analyses 

Assessment  of  individual  and  team  performance  in  realistic  combat  environments  requires  (a)  the 
capability  to  produce  complex  and  dynamic  scenarios,  (b)  identification  of  constructs  that 
represent  important  individual  and  team  skills,  and  (c)  identification  and/or  development  of 


construct  measures.  In  addition,  each  scenario  must  have  identified  quantitative  criteria  of 
mission  success.  Cognitive  task  analyses  informed  and  influenced  the  development  of  scenario 
characteristics  and  the  identification  of  primary  performance  constructs  and  measures. 

Several  cognitive  task  analyses  have  been  performed  on  AW  ACS  operational  personnel.  First, 
analyses  have  been  performed  with  regard  to  AW  ACS  individual  tasks,  with  a  primary  focus  on 
display  enhancements  (Klinger  et  al.,  1993).  In  addition,  Fahey  and  associates  (Fahey  et  al.,  in 
review)  investigated  the  AWACS  task  domain  with  a  special  focus  on  AWACS  team  tasks, 
utilizing  cognitive  task  analysis  and  critical  incident  techniques  for  data  elicitation.  Their  report 
provides  a  comprehensive  description  of  the  AWACS  task,  and  the  general  finding  that  critical 
incident  technique  was  quite  useful  for  the  description  of  AWACS  teamwork. 

In  addition,  a  follow-up  analysis  of  AWACS  tasks,  with  a  primary  focus  on  team  functions  has 
been  articulated  and  initiated  (MacMillan  et  al.,  1998).  This  was  a  preliminary  investigation,  to 
test  a  team-based  approach  to  cognitive  task  analysis.  Results  were  very  informative,  supporting 
frameworks  and  constructs  within  an  information-requirements  approach  to  task  descriptions. 

4.  Articulation  of  AWACS  Teamwork 

One  core  dimension  that  characterizes  teams  in  general  is  the  type  and  degree  of  interdependence 
among  team  members  (Saavedra  et  al.,  1993;  Salas,  et  al.,  1992).  Teams  are  distinguished  from 
groups  in  general  by  a  common  purpose  or  goal,  performed  by  interdependent  team  members 
(Salas  et  al.,  1992).  From  this  definition  we  derived  a  core  definition  of  teamwork:  The 
fundamental  function  of  teamwork  is  the  effective  managing  of  interdependencies  to  accomplish 
a  team  goal.  From  this  core  definition  we  identified  6  dimensions  of  team  interdependence 
(Elliott  &  Schiflett,  in  review). 

Discussion  with  focal  groups  comprised  of  AWACS  subject  matter  experts  led  to  refinement  of 
our  preliminary  team  task  taxonomy,  as  applied  to  AWACS  team  training  and  performance 
evaluation.  Prior  to  the  ROADRUNNER  ‘98  exercise,  we  interviewed  38  expert  AWACS 
weapons  directors.  These  groups  reviewed  the  proposed  scenarios  and  assisted  in  the  evaluation 
and  refinement  of  an  instructor/observer  rating  form  (see  Appendix).  Review  of  the  initial 
taxonomy,  which  was  generated  to  classify  teams  in  general,  led  to  the  identification  of  four 
teamwork  functions  which  are  specific  to  AWACS  teamwork:  (a)  mission  planning,  (b) 
communication  content/timing,  (c)  adherence  to  communication  protocol,  and  (d)  supporting 
behavior. 

4.1  Mission  planning:  formulation  of  “ contracts ” 

AWACS  performance  can  be  divided  into  three  phases:  premission  planning,  task  execution,  and 
debriefing  (discussion  after  task  execution).  Mission  planning  affects  AWACS  teamwork  by 
establishing  roles,  responsibilities,  and  contingency  plans.  As  discussed  by  Fahey  et  al.  (1998) 
and  Macmillan  et  al.  (1998)  AWACS  weapons  directors  explicate  roles,  responsibilities,  and 
strategies  to  manage  team  member  interdependencies  through  the  establishment  of  “contracts”. 
These  contracts  are  made  among  AWACS  team  members  (internal  contracts),  and  to  the 
“external”  team  (i.e.,  pilots  of  friendly  assets). 


4.2  Communication:  adherence  to  protocol 

AWACS  weapons  director  tasks  are  based  predominately  on  the  exchange  of  verbal  information. 
Communications  are  heavily  standardized  in  terms  of  content  (jargon)  and  process.  This  aspect 
of  communication  effectiveness  refers  to  the  degree  to  which  individual  follow  guidelines  for 
communication  exchange.  In  addition  to  proper  jargon  and  syntax,  communications  must  be 
clear,  concise,  and  correct. 

4.3  Communication  to  support  situation  awareness 

Communications  may  be  clear,  concise,  correct,  and  delivered  according  to  proper  protocol,  and 
still  be  superfluous.  The  maintenance  of  situation  awareness  also  requires  that  pertinent 
information  be  exchanged  to  the  right  person,  at  the  right  time.  In  such  a  communication-rich 
environment,  too  much  communication  can  impede  performance,  when  unnecessary  information 
“steps  over”  other,  more  urgent  communication.  Indeed,  part  of  the  proper  timing  of  AWACS 
communications  to  pilots  involves  knowledge  of  when  to  keep  quiet. 

4.4  Supporting  behavior 

While  it  would  seem  apparent  that  this  aspect  is  most  clearly  representative  of  teamwork,  the 
conceptualization  of  supporting  behavior  among  AWACS  weapons  directors  was  more  difficult 
to  validate  through  subject  matter  experts.  While  there  was  no  doubt  that  AWACS 
teammembers  do  support  each  other,  the  argument  was  made  that  most  of  the  support  is  in  the 
form  of  communication  exchange.  AWACS  teammembers  support  each  other  primarily  through 
updates  and  reminders  of  salient  information.  At  the  same  time,  they  can  also  transfer  resources 
(responsibilities)  and  confer  on  decisions/actions,  therefore  it  was  decided  to  keep  this  construct 
as  an  independent  aspect  of  AWACS  teamwork. 

5.  Rating  Instrument 

Once  the  dimensions  of  AWACS  teamwork  performance  were  refined  by  the  SME  focus  groups, 
the  rating  instrument  was  tailored  to  capture  these  dimensions  across  three  phases  of 
performance:  (a)  premission  briefing,  (b)  mission  performance,  and  (c)  mission  debriefing. 
Ratings  were  based  on  a  4-point  scale,  using  traditional  rating  assessment  categories.  The  rating 
categories  used  are  based  on  the  same  categories  of  performance  used  in  AWACS  WD  training. 


AWACS  Team  Observer  Form 
Rating  and  Definition  of  Performance  Criteria 

1.  No  Ability  or  Knowledge.  Task  Failure 

2.  Lacks  Proficiency.  (Coordination,  Communication,  Cohesion). 

3.  Uncorrected  errors.  Degraded  Mission  Outcome  or  Endangered  Friendly  Forces. 

4.  Limited  Proficiency.  Recognizes  and  corrects  errors  with  team  recovery.  Mission 
Degraded. 

5.  Proficient.  No  mission  impacting  errors.  Team  reacts  correctly  in  current  situations. 

6.  Highly  Proficient.  Prevents  errors.  Team  anticipates  future  situations.  Plans  ahead. 

NA  =  Not  Applicable  NO  =  Not  Observed  NP  =  Not  Performed 

Table  1:  Rating  Scale  for  Dimensions  of  Performance 

WDs  were  rated  on  the  following  during  the  Phase  I  portion  of  their  performance  evaluation: 

(a)  Development  of  Mission  Aids, 

(b)  Formulation  of  contracts  internal  to  the  AWACS  team, 

(c)  Formulation  of  contracts  external  to  the  AWACS  team,  and 

(d)  Prebrief  of  pilots. 

These  aspects  were  identified  during  the  subject  matter  expert  focal  interviews  as  critical  to 
successful  premission  planning  and  team  performance.  The  development  of  mission  aids 
included  the  generation,  refinement,  and  review  of  the  communications  worksheet  (which 
specifies  who  talks  to  whom  on  which  channel),  the  fighter  flow  sheet,  the  chart,  and  the  fact 
sheet,  all  of  which  describes  and/or  specifies  procedures  to  enhance  awareness  and  coordination. 
Formulation  of  internal  contracts  refers  to  the  agreements  made  among  the  WDs  as  to  their  roles 
and  responsibilities.  Communication  tasks  and  console  assignments  are  set,  specific  mission 
objectives  are  discussed,  and  functional  responsibilities  and  contingency  plans  are  specified  and 
assigned.  Formulation  of  external  contracts  refers  to  the  procedures  executed  between  the  WDs 
and  others  (pilots)  with  regard  to  roles  and  responsibilities,  such  as  those  regarding  objectives, 
strategies,  and  communications.  These  contracts  are  then  discussed  within  the  briefing  session 
(Pre-brief  Pilots)  by  the  lead  WD  with  the  lead  pilot,  along  with  issues  such  as  the  rules  of 
engagement  (ROE)  and  Air  Tasking  Order  (ATO)  compliance. 

Phase  II  performance  (during  mission  execution)  was  assessed  by  ratings  regarding  (a) 
communication  in  accordance  with  3-1  and  Unit  Standards,  (b)  communications  in  support  of 
situation  awareness  and  the  “big  picture”,  and  (c)  mission  execution.  Communications  are 
assessed  in  terms  of  adherence  to  established  protocol,  which  strives  to  maximize  aspects  of 
clarity,  brevity,  and  accuracy — communications  that  are  clear,  concise,  and  correct.  This 
includes  communications  among  WDS  and  also  their  communications  to  others  (rated 
separately).  In  addition,  communications  are  also  rated  with  regard  to  relevance  and 
timeliness — is  the  information  what  was  needed  to  maintain  team  situation  awareness — was  the 
right  information  “pushed”  to  the  right  person  at  the  right  time?  Mission  execution  was  assessed 


through  consideration  of  ATO  execution,  contract  execution  (formulated  during  premission 
briefing),  and  adaptive  replanning  (flexibility,  problem-solving,  contingency  generation  as 
needed). 

Phase  III  (mission  debriefing)  performance  was  assessed  through  ratings  of  (a)  reconstruction  of 
the  engagement,  (b)  evaluation  of  team  objectives,  (c)  review  of  equipment  issues,  (d)  review  of 
team  mission  execution,  and  (e)  review  of  information  exchange.  Reconstruction  of  the 
engagement  was  assessed  through  consideration  of  the  process  of  reviewing  the  recordings  and 
the  identification  of  conflicts  and/or  problems.  Objectives  were  reviewed  in  light  of  mission 
support  and  execution  of  contracts.  Equipment  issues  included  communication,  console 
assignments,  and  any  failures/alibis.  Team  mission  execution  was  assessed  through 
consideration  of  priority  objectives,  training  objectives,  and  any  failures/alibis.  Information 
exchange  was  considered  with  regard  to  internal  and  also  external  communications. 
Communications  should  have  supported  overall  situation  awareness,  and  lessons  learned  should 
be  identified  during  the  mission  debriefing. 

Each  team  was  rated  on  various  dimensions  of  performance  across  three  phases  of  performance, 
for  each  mission.  They  were  assessed  for  performance  in  Phase  I  (premission  briefing),  Phase  II 
(mission  execution),  and  Phase  III  (mission  debriefing).  Measures  and  variable  names  are  as 
follows: 

5.1  Premission  Briefing 

Development/Use  of  Mission  Aids  (MSNAID_1):  Effective  use  of  appropriate  mission 
planning  aids,  such  as  communications  worksheets,  charts,  fighter  flow  sheet,  and  fact  sheet. 

Negotiation  of  contracts  internal  to  AW  ACS  team  (ICONTR_l):  Effective  discussion 
and  coordination  of  interdependencies  among  AWACS  WDs  and  AST,  with  regard  to 
communications,  console  setup,  mission  tasks  (who  is  primary/assist;  coordination  with  AST), 
and  mission  objectives. 

Negotiation  of  contracts  external  to  AWACS  team  (ECONTR_l):  Effective  discussion 
and  coordination  with  pilots,  with  regard  to  communications,  mission  tasks,  and  objectives. 

Pre-brief  with  Pilots  (PREYI  1).  Effective  discussion  and  coordination  with  pilots,  with 
regard  to  objectives,  standards  (Wing  and  Squadron),  Air  Tasking  Order  compliance,  and  Rules 
of  Engagement. 

Overall  for  Premission  Briefing  (PHASEI).  Average  of  above.  Measures. 

5.2  Mission  Execution 

Communications  in  Accordance  with  standards,  internal  to  AWACS  Teams  (COMMI_2): 
Extent  of  compliance  to  standard  communication  protocols,  regarding  continuum  of  control, 
Land  Unit  Standards,  C-3  (Clear,  Concise,  Correct),  and  use  of  correct  call  signs. 

Communications  in  accordance  with  standards,  external  to  AWACS  Teams! COMME_2): 
Extent  of  compliance  to  standard  communication  protocols,  regarding  continuum  of  control, 
Land  Unit  Standards,  C-3  (Clear,  Concise,  Correct),  and  use  of  correct  call  signs. 

Maintenance  of  Situational  Awareness  and  Big  Picture  in  communications  internal  to 
AWACS  Team(SA  PI  2):  Content  and  timing  of  communications  (as  opposed  to  process). 

Maintenance  of  Situational  Awareness  and  Big  Picture  in  communications  external  to 
AWACS  Team(SA_PE_2):  Content  and  timing  of  communications  (as  opposed  to  process). 


Mission  Execution  internal  to  AWACS  Team  (MSNEXI_2):  Performance  with  regard  to 
execution  of  Air  Tasking  Order,  Contract  execution,  and  adaptive  replanning  (flexibility, 
problem  solving,  and  contingency). 

Mission  Execution  external  to  AWACS  Team  (MSNEXE_2):  Performance  with  regard 
to  execution  of  Air  Tasking  Order,  Contract  execution,  and  adaptive  replanning  (flexibility, 
problem  solving,  and  contingency). 

Overall  mission  execution  (PHASER):  average  of  six  measures  above. 

5.3  Team  Debrief 

Reconstruction  of  engagement-  internal  to  AWACS  Team  (RECENG_3):  Effectiveness 
of  review  of  engagement  through  review  of  recording  and  identification/discussion  of 
conflicts/problems . 

Evaluation  of  Performance  with  regard  to  objectives  internal  to  AWACS  Team 
(EVALON_3):  Effectiveness  of  review  of  performance  with  regard  to  mission  support  and 
contracts. 

Equipment-  internal  to  AWACS  Team  (EQUIP_3):  effective  identification  and 
discussion  of  issues  with  regard  to  equipment,  such  as  communication  and  checkout,  console 
assignments  and  check-outs,  and  failures/alibis. 

Discussion  of  Team  Mission  Execution  external  to  AWACS  Team  (TMISEX  3). 
Identification  and  discussion  of  issues  with  regard  to  support  of  priority  objectives,  WD/AST 
training  objectives,  and  failures/alibis. 

Information  exchange  Internal  to  AWACS  Team  (INFEXI_3)-  Effective  discussion  of 
information  exchange  among  AWACS  team  members,  with  regard  to  maintenance  of  shared 
mental  picture  (situation  awareness)  and  lessons  learned. 

Information  exchange  external  to  AWACS  Team  (INFEXE  3).  Effective  discussion  of 
information  exchange  among  AWACS  team  members  and  others,  with  regard  to  maintenance  of 
shared  mental  picture  (situation  awareness)  and  lessons  learned. 

Overall  Mission  Debriefing  (PHASEIII).  Average  of  six  measures  above. 

Overall  Mission  Performance  (OVERALL).  Average  of  all  measures. 

6.  Results 

Descriptive  Statistics:  Average  ratings  across  teams  (3),  scenarios  (7),  and  raters  (2  per  team  and 
scenario  for  Phase  II).  Results  indicate  variance  in  distribution  in  ratings,  with  utilization  of  the 
entire  rating  scale,  for  most  measures. 


Premission  Planning: 

N 

Minimum 

Maximum 

Mean  Std.  Deviation 

MSNAID_1 

25 

2.00 

4.00 

3.0800 

.4933 

ICONTR_l 

25 

2.00 

4.00 

2.9600 

.6758 

ECONTR_l 

24 

1.50 

4.00 

2.8125 

.6726 

PREVI_1 

24 

1.00 

4.00 

2.8542 

.6833 

PHASE  I 

25 

1.88 

3.75 

2.9200 

.4703 

Task  Execution: 

COMMI_2 

42 

1.00 

4.00 

2.9881 

.7690  ! 

COMME_2 

42 

1.00 

4.00 

2.7143 

.7741 

SA_PI_2 

42 

1.00 

4.00 

2.9524 

.7949 

SA_PE_2 

42 

1.00 

4.00 

2.6905 

.8762 

MSNEXI_2 

42 

1.00 

4.00 

3.0000 

.7730 

MSNEXE_2 

41 

1.00 

4.00 

2.6951 

.9413 

PHASE  II 

42 

1.33 

4.00 

2.8421 

.7301  i 

Debrief: 

RECENG_3 

24 

2.00 

4.00 

3.1250 

.6124 

EVALON_3 

24 

2.00 

4.00 

3.0417 

.7929 

EQUIP_3 

26 

2.00 

4.00 

3.1154 

.4961 

TMISEX_3 

24 

1.00 

4.00 

3.0833 

.8805 

INFEXI_3 

25 

1.00 

4.00 

3.0400 

.8406 

INFEXE_3 

24 

1.00 

4.00 

2.9792 

.8140 

PHASE  III 

27 

1.50 

4.00 

3.0562 

.5970 

OVERALL 

22 

1.94 

3.69 

2.8699 

.4705 

Valid  N  (listwise) 

16 

6.1  Differences  in  performance  attributable  to  differences  among  raters. 

ANOVA  analyses  indicated  no  significant  differences  among  raters  with  regard  to  any  of  the 
measures  (See  Appendix  for  descriptives  and  ANOVA  results).  In  addition,  regression  analyses 
based  on  the  consideration  of  trusted  agents,  teams,  and  scenarios  indicate  that  the  variance 
attributable  to  differences  among  trusted  agents  was  not  significant. 

6.2  Relative  impact  of  three  sources  of  variance  (trusted  agents,  teams,  and  scenarios ). 
There  were  three  primary  sources  of  variance:  that  attributable  to  raters  (which  should  be 
minimized),  to  differences  among  teams  (not  of  primary  interest  here),  and  to  the  training 
exercise  itself  (differences  among  the  7  scenarios  was  predicted  to  demonstrate  improvement 
over  time).  Multiple  regression  analyses  were  performed,  first  on  the  model  composed  of  trusted 
agents,  teams  and  scenarios.  This  model  predicted  73%  of  the  total  variance  in  ratings.  In 
addition,  the  consideration  of  trusted  agents  did  not  add  to  the  prediction  of  ratings,  thus 
indicating  reliability  of  measures  with  regard  to  raters.  The  analyses  were  then  run  using  teams 
and  scenarios  as  predictors,  and  the  model  still  predicted  73%  of  the  total  variance  in  ratings. 


Model  Summary:  Predictors:  (Constant),  SCENARIO,  TEAM,  TRUSTEDA 


Dependent  Variable:  OVERALL 


Model  R 

R  Square 

Adjusted  R  Square 

Std.  Error  of  the  Estimate 

1  .859 

.738 

.694 

.2603 

ANOVA  Sum  of  Squares 

df 

Mean  Square 

F 

Sig. 

1  Regression 

3.429 

3 

1.143 

16.867 

.000 

Residual 

1.220 

18 

6.776E-02 

Total  4.648 

21 

Coefficients 

Coef-B 

S.Err 

Beta 

t 

Sig. 

1  (Constant) 

2.591 

.221 

11.750 

.000 

TRUSTEDA 

-4.394E-02 

.045 

-.117 

-.969 

.345 

TEAM 

-.170 

.070 

-.294 

-2.436 

.025 

SCENARIO 

.177 

.027 

.792 

6.557 

.000 

a  Dependent  Variable:  OVERALL 

These  data  indicate  that  participation  in  the  training  exercise  had  a  significant  effect  on 
performance,  across  all  teams,  and  for  most  measures  of  performance  across  the  three  phases  of 
performance.  Results  are  described  in  further  detail  for  the  effects  due  to  differences  among 
teams,  and  differences  across  scenarios. 

6.3  Differences  in  performance  attributable  to  differences  among  teams. 

Data  demonstrate  the  advantage  in  careful  construction  of  performance  constructs.  The  overall 
measure  of  performance  was  based  on  the  mean  of  all  measures;  ANOVA  results  did  not  indicate 
significant  differences  among  teams.  There  were,  however  differences  among  teams  for  the 
measurement  of  one  aspect  (negotiation  of  contracts  among  team  members)  of  performance 
during  premission  briefing,  and  for  nearly  all  aspects  of  performance  during  mission  execution. 
Teams  did  not  differ  significantly  in  performance  during  mission  debriefing.  An  overall 
assessment  of  performance  may  be  too  generic  to  capture  effects  of  a  predictor  variable. 

ANOVA:  Significant  Differences  By  Team,  Between  Groups 


Variable 

Sum  of  Squares 

df 

Mn  Sq 

F 

Sig.  | 

ICONTR_l 

3.363 

2 

1.681 

4.869 

.018** 

MSNEXE_2 

5.730 

2 

2.865 

3.665 

.035**  | 

SA_PE_2 

4.298 

2 

2.149 

3.083 

.057* 

SA  PI  2 

4.762 

2 

2.381 

4.392 

.019** 

COMME_2 

3.857 

2 

1.929 

3.631 

.036** 

COMMI_2 

2.869 

2 

1.435 

2.617 

.086* 

PHASE2 

3.298 

2 

1.649 

3.466 

.041** 

Performance  Ratings  for  Phase  I:  By  Team 
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Performance  Ratings  for  Phase  II:  By  Team 
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Performance  Ratings  for  Phase  III:  By  Team 


6.4  Differences  attributable  to  scenarios  and  experience. 

Team  performance  differed  significantly  across  scenarios  for  many  of  the  measures  (See 
Appendix  for  descriptives  and  ANOVA  output).  Differences  in  measures  among  scenarios 
would  capture  differences  due  to  the  scenario  content,  as  scenarios  differed  in  mission  objective 
and  content.  They  would  also  be  due  to  experience  over  time.  However,  Scenario  1  and 
Scenario  7  are  very  similar  in  content,  to  better  ascertain  the  effect  of  experience  on 
performance.  First  we  describe  differences  across  all  scenarios.  The  following  graph  describes 
overall  results  of  instructor  ratings  of  AW  ACS  team  performance,  for  each  mission  session, 
ordered  over  time. 


4.0 


3.5 

3.0 

2.5 

2.0 

1.5 

1.0 

.5 

0.0 


2  3  4  5 

SCENARIO 


■  phasei 

I  I PHASE2 

■  PHASE3 
□  OVERALL 


The  above  graph  provides  a  succinct  glimpse  of  trends  regarding  the  effect  of  simulation 
participation  on  performance,  through  differences  in  performance  from  Scenario  1  and  Scenario 
7,  by  phase.  Here  it  can  be  seen  that  variance  in  performance  across  scenarios  was  more 
attributable  to  performance  in  Phases  II  and  III:  teams  were  lower  in  performance  during  Phase 
II  and  III  during  the  first  scenario,  and  demonstrated  higher  improvement  in  these  phases.  At  the 
same  time,  performance  during  Phase  I  (premission  briefing)  also  improved  over  time. 

Overall  results  for  each  team  for  Scenario  1  (pretest)  and  Scenario  7  (post-test).  Team  A  was  the 
highest  performing  team  for  both  pre-and  post  scores.  Improvement  for  this  team  was  consistent 
across  the  three  phases.  In  comparison,  Teams  B  and  C  performed  less  well  from  the  start,  and 
particularly  less  well  in  Phase  II  (mission  performance)  and  Phase  III  (debriefing  performance). 
Data  indicate  that  training  needs  exist  for  the  kind  of  training  that  distributed  mission  training 
provides:  improvement  in  team  premission  briefing,  team  coordination  and  team  debriefing.  In 
addition,  data  demonstrate  the  effectiveness  of  participation  in  the  training  exercise  for  the 
improvement  of  performance  across  all  phases  of  each  mission. 

7.  Discussion 

Results  were  consistent  with  expectations.  Simulation-driven  distributed  mission  training  were 
expected  to  add  value  through  the  experience  of  complexity  of  team-on-team  interdependencies. 
The  immersion  of  individuals  into  teams  within  a  global  mission  scenario  should  enhance  their 
capability  of  managing  these  interdependencies  during  performance.  Debriefing  experience  is 
particularly  valuable  to  AW  ACS  crew  as  they  often  do  not  get  the  advantage  of  debriefing  with 
pilots  after  the  mission;  their  mission  usually  extends  beyond  the  time  of  performance  of  a 
particular  combat  aircraft.  The  value  of  this  experience  is  indicated  by  the  increased 


performance  during  debriefing  of  later  mission  scenarios.  In  addition,  teams  improved  on  most 
aspects  of  performance  throughout  mission  planning  and  execution. 

Data  demonstrate  the  advantage  of  careful  construction  of  performance  constructs.  The  overall 
measure  of  performance  was  based  on  the  mean  of  all  measures;  ANOVA  results  did  not  indicate 
significant  differences  among  teams.  There  were,  however  differences  among  teams  for  the 
measurement  of  one  aspect  (negotiation  of  contracts  among  team  members)  of  performance 
during  premission  briefing,  and  for  nearly  all  aspects  of  performance  during  mission  execution. 
Teams  did  not  differ  significantly  in  performance  during  mission  debriefing.  An  overall 
assessment  of  performance  may  be  too  generic  to  capture  effects  of  a  predictor  variable. 

8.  Summary 

The  following  quote  very  aptly  captures  the  essence  of  the  ROADRUNNER  ’98  experience: 

The  ROADRUNNER  ‘98  concept  of  operations  and  its  execution  was  a  valuable  tool 
in  development  and  evolution  of  Distributed  Mission  Training  (DMT).  This  exercise 
demonstrated  the  ability  to  interface  multiple  virtual  warfighters  and  synthetic  forces 
to  form  a  virtual  battlespace  that  can  be  used  to  conduct  realistic  combat-oriented 
training.  Our  goal  was  to  compare  team  performance  before  and  after  DMT  training 
and  to  give  the  participants  the  opportunity  to  critique  both  the  systems  and  this 
concept  of  training.  The  initial  comments  from  the  AWACS  crews  suggest  that  they 
found  DMT  to  be  a  viable  and  productive  training  tool.  The  technology  was  proven 
capable  of  providing  combat-oriented  training  at  low  cost,  with  fewer  risks,  while  still 
maintaining  the  security  constraints  associated  with  aviation  duties.  ROADRUNNER 
‘98  has  shown  the  potential  to  provide  improvements  in  pre-planning  and  mission 
execution  with  emphasis  in  shared  training  exercises  to  provide  endless  training 
opportunities  to  the  21st  century  aviator. 

Randy  L.  Parton,  Major,  USAF 
Chief,  Modeling  and  Simulation 

The  approach  taken  during  the  evolution  of  the  rating  scales  enabled  the  collection  of  data  that 
significantly  adds  to  the  understanding  and  research  of  team  communications,  interdependencies, 
and  training. 

Post-mission  questionnaires  provided  comments,  information,  and  guidance  that  have  proves 
helpful  in  the  creation  of  continued  DMT  missions  and  research. 


Participants  responded  to: 

How  mission  impacted  learning  and  training: 

•  A  staged  4-day  war  would  be  very  beneficial. 

•  Using  voice/data  recording  capability  helps  us  pinpoint  errors  &  allows  a  thorough 
debrief,  enhancing  the  learning  experience. 

Whether  exercise  helped  meet  squadron  training  objectives: 

•  Good  intro  to  a  Red  Flag  exercise. 

•  Great  real  war  training,  with  inexperienced  ...  benefiting  most. 

Whether  exercise  has  the  potential  of  meeting  squadron  training  objectives: 

•  Work  the  bugs  and  this  could  be  a  simulated  Red  Flag. 

•  It  will  definitely  increase  amount  of  learning  gained  from  SIM. 

Suggestions  for  improvements: 

•  Tie  SIMs  together  with  a  grand  order  of  battle,  Intel,  and  more  complete  mission 
planning. 

•  Allow  full- crew  participation. 

•  Anything  that  can  go  wrong  on  the  jet  needs  to  be  able  to  go  wrong  here. 

In  addition,  participants  commented  on  the  number  of  “lessons  learned”  that  occur  when  training 
involves  so  much  training  in  one  week,  and  believed  the  non-predictability  of  real  pilots  manning 
the  aircraft,  as  opposed  to  sim  drivers,  added  indescribably  to  the  realism  of  the  mission. 

The  ongoing  development  of  distributed  mission  training  will  allow  USAF  to  move  into  the  next 
century  as  a  cohesive,  well  trained  warfighting  unit.  The  technology  and  research  providing  the 
foundation  for  this  capability  will  allow  for  a  more  efficient,  cost-effective,  and  objective  means 
of  training  that  has  only  begun  to  make  its  impact  on  the  military  concept  of  warfighter 
readiness. 

9.  References 

[Elliott,  in  review]  Elliott,  L.R.  &  Schiflett,  S.  (in  review).  Development  of  Synthetic  Team 
Training  Environments:  Application  to  USAF  Command  and  Control  Aircrews.  To  appear  in  H. 
O’Neil  &  D.  Andrews  (Eds.)  Aircrew  Training:  Methods,  Technologies,  and  Assessment. 

[Fahey,  in  review]  Fahey,  R.P.,  Rowe,  A.,  Dunlap,  K.  and  DeBoom,  D.(in  review).  Synthetic 
Task  Design  (1):  Preliminary  Cognitive  Task  Analysis  ofAWACS  Weapons  Director  Teams. 
Technical  Report.  Brooks  AFB,  TX:  Armstrong  Laboratory. 

[Fogleman,  1996]  Fogleman,  General.  Chief  of  Staff  Air  Force:  Perspective.  29  Jan  96. 

[Klinger,  1993]  Klinger,  D.W.,  Andriole,  S.J.,  Militello,  L.G.,  Adelnam,  L.,  Klein,  G.  &  Gomes, 
M.E.  Designing  for  performance:  A  cognitive  systems  engineering  approach  to  modifying  an 
AWACS  human-computer  interface.  AL/CF-TR- 1993-0093.  Wright-Patterson  AFB,  OH: 
Armstrong  Laboratory,  1993. 


[MacMillan,  1998]  MacMillan,  J.,  Serfaty,  D.,  Young,  P.,  Klinger,  D.,  Thordsen,  M.,  Cohen, 

M.,  Freeman,  J.  &  Elliott,  L.R.  A  system  to  enhance  team  decisionmaking  performance:  Phase  1 
Final  Report.  Brooks  AFB,  TX:  AFRL  Warfighter  Training  Research  Division.  1998. 


[Parton,  1998]  Parton,  Randy.  ROADRUNNER '98  After  Action  Report.  7  Aug  98. 


[Saavedra,  1993]  Saavedra,  R.,  Earley,  P.C  &  Van  Dyne,  F.  Complex  interdependence  in  task¬ 
performing  groups.  Journal  of  Applied  Psychology ,  78  (1),  61-72,  1993. 


[Salas,  1992]  Salas,  E.,  Dickinson,  T.,  Converse,  S.  &  Tannenbaum,  S.  Toward  and 
understanding  of  team  performance  and  training.  In  R.  Swezey  &  E.  Salas  (Eds.)  Teams:  Their 
training  and  performance .  Norwood  NJ:  Ablex  Publishing  Corp.  1992. 


